Ensemble Methods for Handwritten Digit Recognition

Neural network ensembles are applied to handwritten digit recognition. The invidual networks of the ensemble are combinations of sparse Look-Up Tables with random receptive fields. It is shown that the consensus of a group of networks outperform the best invidual of the ensemble and further we show that it is possible to estimate the ensemble performance as well as the learning curve, on a medium size database. In addition we present preliminary analysis of experiments on a large data base and show that state o f t h e ad performance can be obtained using the ensemble approach by optimizing the receptive fields. INTRODUCTION Recognition of handwritten digits is a serious, current candidate for a “real world” benchmark problem to assess pattern recognition methods: For a recent review see [4]. It has been the object of a recent state of the art application of neural networks [5 ] . Neural network ensembles were introduced recently as a means for improving network training ‘and performance. The consensus of a neural network ensemble may outperform individual networks [I] and ensembles can be used to implement oracle functions [2]. Furthermore the consensus may be used for realization of fault tolerant neural network architectures [3]. Within the present system for recognition of handwritten digits, we find that the ensemble consensus outperform the best individual of the ensemble by 20 25%. However, due to correlation among errors made by the participating networks, the marginal benefit obtained by increasing the ensemble is low once the ensemble size is 2 15. Our findings are in line with the results obtained in [l]. We illustrate the theoretical tools for predicting the performance of the ensemble consensus, and we demonstrate the use of the ensemble as an oracle in its capacity of predicting the learning curve, ie. the number of test errors as a function of the number of training examples. This and other ensemble oracle functions were introduced by Salamon e2 al. [2]. Real world applications face the problem of ‘CO.Y?IE:C‘I, Electronics Institute B349, The Technical University of Denmark, DK-2800 ’ C O Y ~ E C ‘ I , Dept. of Optics and Fluid Dynamics, Ria National Laboratory, DK-4000 3Dept. of Mathematical Sciences, San Diego State University, San Diego CA 92182 USA, Lyngby Denmark, [email protected] Roskilde, [email protected] [email protected] 333 @7803-0557-4/92$03.~